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The rare top quark couplings tu^ and tu^y (tt; = u, c) induced at the one-loop level by a flavor 
violating tmH vertex are studied within the context of an effective Yukawa sector that incorporates 
SUl{2) x Uy (l)-invariant operators of up to dimension six. Data on the recently observed D° — D° 
mixing are employed to constrain the tUiH vertex, which is then used to predict the t — > UiH, 
t — 5* Ui*y, and t — > it,:77 decays, as well as the 77 — > tiH + tUi reaction in the context of the ILC. It 
is found that the t — > cH and t — > C77 decays can reach sizable branching ratios as high as 5 x 10 -3 
and 10 -4 , respectively. As for the t — ¥ C7 decay, it can have a branching ratio of 5 x 10 -8 that is 
about 6 orders of magnitude larger than the standard model prediction, which, however, is still very 
small to be detected. As for tc production, it is found that, due to the presence of a resonant effect 
in the convoluted cross section a(e + e~ — > 77 — > tc + ic), about (0.5 — 2.7) x 10 3 tc events may be 
£N| ] produced at the ILC for a value of the Higgs mass near to the top mass. 

' The fact that the top quark is the heaviest elementary particle known suggests that it would be quite sensitive to 
new physics effects. In the standard model (SM), it decays almost exclusively into the bW mode. Although some 
O ,■ of the treedevel t —> diW (di — d, s), t — » diWZ, and t —> UiWW [l] (ui = u, c) decays can have sizable branching 
ratios, they are indeed suppressed. At the one-loop level, there arise the interesting flavor violating decays t —> UiV 
(V — g,y,Z,H) 0, ||, t — > egg [U, and t —> ccc @, which are however considerably GIM-suppressed. Although very 
suppressed in the SM, the flavor violating t — > UiV decays can have sizable branching ratios in many well motivated 
£C) . of its extensions, such as the two-Higgs doublet model 51,1 31, su persymmetric models (jlHI, some exotic scenarios [lof . 
£> ' 331 models [ll|, and model-independent descriptions [12.Tl3j. On the other hand, it is expected that new physics 
effects on top qua rk p hysics can be observed in both LHC and ILC colliders, since they complement the capabilities 
of each other [lH Il5j |. In addition the ILC may be operated in the 77 collision mode, which will provide a cleaner 
environment for top quark physics (T^ |. 

In a recent communication by some of us [l6l |. exact amplitudes for the one-loop induced fifj^f and fifj'yy couplings, 
with a fi a quark or charged lepton, induced by a nondiagonal fifjH vertex was presented in the context of new sources 
of flavor violation arising from extended Yukawa sectors. An effective Lagrangian description for the SM Yukawa sector 
was proposed, which was extended to include dimension-six SUl(2) X £/y-(l)-invariant operators that generate a general 
flavor and CP-violating fifjH coupling of renormalizable type without the necessity of including additional degrees of 
freedom. Applications to lepton flavor violation were done in [l6| . In this paper, we will apply these general results to 
study the rare top quark decays t UiH, t —¥ 1*^7, and t — ¥ 1*^77, as well as to investigate the 77 — > tUi + tv,i reaction 
in the context of the ILC operating in the 77 mode, which has shown to be the most efficient mechanism in producing 
tc events, as the corresponding cross section can be up to 2 orders of magnitude and 1 order of magnitude greater than 
those associated with the production mechanism in the e + e~ and e^y collision modes, respectively [17-24]. In fact, the 
possibility of detecting tc events at linear e + e~ colliders operating in the e + e~, ej, and 77 modes has been considered 
by some authors in diverse contexts. In the SM, there is no signal of tc events as the three collision modes have very 
small cross sections: a(e+e- tc) « lCr 10 fb, cr(e~7 -> e~tc) » lCT 9 fb, and (7(77 -> tc) « 10 _8 fb [13, Hj. In 
contrast, cross sections several orders of magnitude larger arise within the context of the general two-Higgs doublet 
model (THDM-III), it was found that a(e + e~ -!• v e v e tc) « 9fb [H[ and (7(77 -> tc) « 0.11 fb [n|. On the other 
hand, the cross sections <r(e + e~ — > tc) ~ 0.02 fb, cr(e~7 — > e~tc) w 0.04 fb, and 17(77 ~~ ► t°) ~ 0.7 fb were reported in 
the context of supersymmetric models @, [2l|. In top-color-assisted technicolor models, the following cross sections 
were found [Hj]: a(e + e~ -> 7*7*, ^*Z* e + e~tc) w 10 _1 fb, a(e~y -> e~tc) w 3fb, and a (77 -> tc) ps 50 fb. 
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More promising results have been found in the context of the littlest Higgs model. In Ref. [23[ the cross sections 
cr(e + e~ — > tc) rj 16 fb, cr(e + e _ — > v e v e tc) ps 10 _3 fb a(e~j — > e~tc) ps 10~ 2 fb were found. More recently, the very 
large cross sections a(e + e~ — > tc) pa 10 4 fb and 17(77 — ^ tc) pa 10 s fb were reported in the context of the littlest Higgs 
model with T-parity 12411 . The capabilities of model- independent tc production at high energy e + e~ colliders have 
been studied in Refs. [25| . Below, we will see that the flavor violating tcH coupling induces an observable cross section 
for the complete e + e~ — > 77 — > tc + tc process. On the other hand, experimental analysis on tc events mediated by a 
vector particle as the photon or the Z gauge boson has been carried out within the context of LEPII and HERA [26| . 
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Constraints on the tc^f and tcZ couplings of order of about 10 _1 were derived [26| . 

In the SM the Yukawa sector is both flavor-conserving and CP-conserving, but these effects can be generated at 
the tree level if new scalar fields are introduced. One alternative, which does not contemplate the introduction of 
new degrees of freedom, consists in incorporating into the classical action the virtual effects of the heavy degrees by 
introducing SUl(2) x [/y(l)-invariant operators of dimension higher than four [131 ] - Indeed, it is only necessary to 
extend the Yukawa sector with dimension-six operators to induce the most general coupling of the Higgs boson to 
quarks and leptons. A Yukawa sector with these features has the following structure [l3T ] 



C eff = ~Yij(Q<**<ti) ~ J-( $t$ )(&<M;) + (!) 

where Yy, Qi, $ 9 (& q = $,$ for q = d,u respectively and a sum over q is implied), di, and Ui stand for the 
usual components of the Yukawa matrix, the left-handed quark doublet, the Higgs doublet, and the right-handed 
quark singlets of down and up type, respectively. The otij numbers are the components of a 3 x 3 general matrix, 
which parametrize the details of the underlying physics, whereas A is the typical scale of these new physics effects. 
This effective Yukawa sector induces a flavor and CP violating coupling fifjH of renormalizable type given by 
FfifiH = -i{u%P R + u[P L ), where uj% = ^6 iS + fl« and w« = + f^-, with P R , L = (1 ± 75 )/2. In these 

expressions, Q( u ' d '> = f^J a^ u, ''^V^ l ' d ^ , being y^" ,d - ) and vj^' d ^ the usual unitary matrices, which correlate 

gauge states to mass eigenstates. Let us emphasize that this vertex describes the most general renormalizable coupling 
of a scalar field to pairs of fermions, which reproduces the mean features of most of extended Yukawa sectors [la ]. 

The above fifjH vertex induces the flavor violating vertices fifj"f and /i/j77 at the one-loop level. The contribution 
to /i/j77 occurs through two set of Feynman diagrams, each of them giving a finite and gauge invariant contribution. 
One of these sets, whose contribution is marginal, include box diagrams, reducible diagrams characterized by the one- 
loop /i/j7 coupling, and reducible diagrams composed by the one-loop fi — fj bilinear coupling (see Fig. 1 of Ref. [l6j). 
The other set of diagrams, which gives the dominant contribution, is characterized by the SM one-loop jjH* coupling, 
with * stands for a virtual Higgs. The exact expression for the amplitude associated with the fifj"f"f vertex with all 
particles on the mass shell is given in Ref. [Hj]. In the same reference, an exact expression for the decay width of the 
fi fjl transition is also presented. We will use these results to study the decays t — ¥ Uij and t — ¥ U,77, as well as 
the scattering process 77 — > tin (see Fig. [1]). As already mentioned, we are also interested in studying the t — > UiH 
decay, whose branching ratio can be written as Br(t — > Ujif) = (Clf u . / (32w))(mt/Tt)(l — {mH/m t ) 2 ) 2 . In writing the 
above expression, we have neglected the m Ui mass. The expression for the branching ratio associated with the t — > C77 
decay can be obtained from the exact expression given in Ref. [l6j for the /i/j 77 coupling. We will consider only 
the contribution arising from the diagrams characterized by the one-loop 77-ff* coupling, as it dominates. In order 
to make predictions, some value for the VLt Ui parameter must be assumed. In Ref. [13| . the branching ratios for the 
t — > u,iV decays were estimated by adopting the Cheng-Sher ansatz slightly modified by introducing the new physics 
scale A instead of the Fermi one v. Qtu, = At Ui «/rnim^~/A. By assuming X tc ~ 1, branching ratios for the t — > cH and 
t — > C7 transitions of the order of 10 -4 and 10 -8 , respectively, were found. On the other hand, the branching ratio 
for the t — > C77 decay was calculated within the context of the THDM-III, which has the simplest extended Higgs 
sector that naturally incorporates Higgs mediated flavor changing neutral current at the tree level [27|. Using the 
Cheng-Sher ansatz, it was found that this decays can reach a branching as high as 10 -4 . In this work, we will support 
our predictions using the available experimental data to get a bound on the tmH vertex. We have found that the best 
constraint [28| arises from the recently observed D° — D° mixing [29l - [3~fl ] . Since this observable receives contributions 
simultaneously from both the tuH and tcH couplings, it is only possible to obtain a bound for the product of the 
two parameters characterizing these vertices, namely, |f2 tc f2 tt j| < 10 -3 . So, in order to make predictions, some extra 
assumption must be introduced. We will assume that fi t „ = lO" 1 ^, which implies the bounds VL 2 C < 10~ 2 and 
Q 2 U < 10~ 4 . It should be mentioned that the experimental data on D° — D° mixing was used in Ref. [32[ to constrain 
this product of couplings within the context of the THDM-III. Although this reference presents a bound for the 
Cheng-Sher parameters (Ay in our notation), we have verified that our bound for |f2 tc f2 t „| coincides essentially with 
the one presented by these authors in the Higgs mass range 100 GeV < m# < 200 GeV. However, we would like to 
stress that our parametrization of the Hfifj coupling is more general than that generated by the THDM-III model 
in the sense that it includes both scalar and pseudoscalar components. Below, we will present results only for those 
processes involving the quark c, which can be translated to the corresponding transitions involving the quark u simply 
multiplying by a factor of 10~ 2 . We will present predictions using the maximum allowed value for the f2t c parameter. 
In Fig. [21 the branching ratio for the t — > cH decay and for the H —> 77 decay are shown as a function of the 
Higgs mass. From this figure, we can see that Br(t — > cH) ranges from 5 x 10~ 3 to 5 x 10~ 4 for a Higgs mass in 
the intermediate range 115 GeV < mu < 2mw, which is of the same order of magnitude that the branching ratio 
for the H — > 77 decay. This prediction is about 1 order of magnitude above than that obtained in Ref. [l3[. On the 
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FIG. 2: The branching ratios for the t — > cH, H —¥ 77, t — > 07, and t — > C77 decays as a function of the Higgs mass. The thin 
lines correspond to a Higgs mass in the range 115 GeV < ran < 171.3 GeV. 

other hand, from this figure we can see that the branching ratio for the t — > 07 decay is indeed suppressed, as it has 
a value essentially constant of about of 5 x 1CP 8 , which however is about 6 orders of magnitude larger than the SM 
prediction. In contrast, the t — ¥ C77 decay can has a significant branching ratio in the intermediate Higgs mass range, 
which can be as high as 10~ 4 — 1CP 5 . This prediction is of the same order of magnitude as that obtained in Ref. [3|- 
It should be noticed that Br(t —> C77) ~ Br(t —> cH)Br(H — > 77), which is evident from graphs in Fig. [5J 

We now turn to discuss the 77 — > tc + tc process in the context of the ILC. From the amplitude for the fifjJJ 
coupling given in Ref. (l6j , one can construct the unpolarized cross section of the 77 — > tc + tc collision, which in turn 
defines the cross section of the complete process e + e~ — > 77 — > tc + tc through the relation 

f Vma * dC 

a{s)= / dz —^-a(jj ^ tc + tc), (2) 

J {m c +mt) I \fs ^ 

where z — yS/y/s, being y/s and y§ the c.m.s. energies of the e + e~ and 77 collisions, respectively. In the above 
expression dJ ~ d 1 z 1 represents the photon luminosity [HJ]. The optimum value for the upper limit in the above integral 
is Umax ~ 0.83. As in the case of the t — > C77 decay, we are considering only the contribution given by the diagrams 
characterized by the SM 77^* vertex. Initially, the ILC will operate from a y/s of 200-500 GeV, with a luminosity 
of 500 ftT 1 within the first years of operation and 1000 fb 1 during the second phase of operation at 500 GeV. After 
operating below or at 500 GeV for a number of years, the ILC could be upgraded to a higher energy [l4|. We will 
present results for energies in the range 250 GeV < y/s < 1000 GeV. In Fig. |3l both the pure and the convoluted cross 
sections are shown as a function of the Higgs mass for some values of vl and t/s. It can be appreciated a significant 
enhancement of the convoluted cross section for values of the Higgs mass near the value of the top mass. This peculiar 
effect, which, as it can be appreciated from the graphic in Fig. |31 is not present in the pure (7(77 — > tc + tc) cross 
section, has its origin in a resonance that occurs for a Higgs mass near the top mass value. To see this, note that 
the convoluted cross sections are proportional to the Higgs propagator ((sz 2 — m 2 H ) 2 + m^T^) -1 , which reach its 
maximum value for z = mn/y/s- On the other hand, from Eq. ([2]) it can be seen that (mt + m c )/y/s < z < y m ax, 
which shows that the integration variable z can be equal to m t /\/s and thus a resonant effect arises when m# takes 
values near the top mass. The importance of the convoluted cross section for a Higgs mass of the order of the top 
mass is best illustrated by showing some specific values. For example, at a y/s of 500 GeV the convoluted cross section 
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FIG. 3: The cross sections for the 77 — > tc + ic and e + e~ — > 77 — > tc + ic processes as a function of the Higgs mass for some 
values of the c.m.s energy. 



takes the values 0.46 fb, 2.72 fb, 2.46 fb, 2.09 fb, and 1.70 fb for a Higgs mass equal to m u 184 GeV, 200 GeV, 210 GeV, 
and 220 GeV, respectively. The maximum value of the cross section corresponds to mj? = 184 GeV. The respective 
number of events is 460, 2720, 2460, 2090, and 1700 if a luminosity of 1000 fb -1 is assumed. From the discussion 
given at the beginning of the paper, we can see that our results for ttih near rat are about 1 order of magnitude 
larger than those derived from the THDM-III [3, [lj| and are lightly above of those predicted in supersymmetric 
models 0, [2l[. However, our results are 1 order of magnitude lower than those derived within the context of top- 
color-assisted technicolor models 22J and several orders of magnitude lower than those obtained in the littlest Higgs 
model [23|, [24|. Our results are conservative since we have used the one- loop SM .H77 coupling. 

In conclusion, in this paper we have investigated some phenomcnological implications of a flavor violating tcH vertex 
induced by an extended Yukawa sector that incorporates S , J7l(2) x Uy (l)-invariant operators of up to dimension six. 
By bounding the tcH vertex from the recently observed D° — D° mixing, the t — > cH, t — > 07, t — > C77, and 
77 —¥ tc + ic processes were predicted. Sizable branching ratios for the t — ¥ cH and t — ¥ C77 decays of the order of 
10 -3 and 10 -4 , respectively, were found. A branching ratio of about 5 x 10~ 8 for the t — > cj decays was found, which 
though it is about 6 orders of magnitude larger than the SM prediction, it is not large enough to be detected. As far 
as the tc production at the ILC is concerned, it was found that about of (0.5 — 2.7) x 10 3 tc events may be produced 
for a Higgs boson with mass near to the top quark mass. 

We acknowledge financial support from CONACYT and SNI (Mexico). 



[1] E. Jenkins, Phys. Rev. D56, 458 (1997). 

[2] J. L. Di'az-Cruz et al, Phys. Rev. D41, 891 (1990); G. Eilam et al, ibid. 44, 1473 (1991); 59, 039901 (1998). 

[3] B. Mele and S. Petrarca, Phys. Lett. B435, 401 (1998). 

[4] G. Eilam et al, Phys. Rev. D73, 053011 (2006). 

[5] A. Cordero-Cid et al, Phys. Rev. D73, 094005 (2006). 

[6] W. S. Hou, Phys. Lett. B296, 179 (1992); M. Luke and M. J. Savage, ibid. 307, 387 (1993); D. Atwood et al, Phys. Rev. 

Lett. 75, 3800 (1995); Phys. Rev. D53, 1199 (1996); 55, 3156 (1997); E. O. Iltan, ibid. 65, 075017 (2002); E. O. Iltan and 

I. Turan, ibid. 67, 015004 (2003); S. Bejar et al, Nucl. Phys. B600, 21 (2001). 
[7] J. L. Di'az-Cruz et al, Phys. Rev. D60, 115014 (1999). 

[8] C. S. Li et al, Phys. Rev. D49, 293 (1994); J. M. Yang and C. S. Li, ibid. 49, 3412 (1994); G. Couture et al, ibid. 52, 
1713 (1995); G. Couture et al, ibid. 56, 4213 (1997); J. L. Lopez et al, ibid. 56, 3100 (1997); G. M. de Divitiis et al, 
Nucl. Phys. B504, 45 (1997); J. Cao et al, ibid. 651, 87 (200 3); J. J. Liu et al , Phys. Lett. B599, 92 (2004); J. Guasch 
and J. Sola, Nucl. Phys. B562, 3 (1999); S. Bejar et al, arXiv |hep-ph/0101294| 

[9] J. M. Yang and C. S. Li, Phys. Rev. D49, 3412 (1994); J. M. Yang et al, ibid. 58, 055001 (1998); G. Eilam, et al, Phys. 
Lett. B510, 227 (2001). 

[10] C. Yue et el, Phys. Lett. B508, 290 (2001); J. A. Aguilar-Saavedra and B. M. Nobre, ibid. 553, 251 (2003); G. Lu et al, 

Phys. Rev. D68, 015002 (2003); R. Gaitan et al, ibid. 72, 034018 (2005). 
[11] A. Cordero-Cid et al, Phys. Rev. D72, 057701 (2005). 
[12] F. del Aguila et al, Phys. Lett. B492, 98 (2000); JHEP 0009, 011 (2000). 



5 



A. Cordero-Ci d et al, Phys. Re v. D70, 074003 (2004). 

J. Brau et al, arXiv:0712.1950l 

G. Weiglein et al, Phys. Rept. 426, 47 (2006). 

J. I. Aranda et al, Phys. Rev. D79, 093009 (2009). 

C.-S. Huang et al, Phys. Lett. B452, 143 (1999). 

W.-S. Hou et al, Phys. Rev. D56, 7434 (1997). 

Jiang Yi et al, Phys. Rev. D57, 4343 (1998). 

Z. H. Yu et al, Eur. Phys. J. C16, 541 (2000). 

J. Cao et al, Nucl. Phys. B651, 87 (2003). 

J. Cao et al, Eur. Phys. J. C41, 381 (2005). 

C.-X. Yue et al, Phys. Lett. B624, 39 (2005). 

X. Wang et al, Nucl. Phys. B807, 210 (2009). 

V. F. Obraztsov et al., Phys. Lett. B426, 393 (1998); T. Han and J. L. Hewett, Phys. Rev. D60, 074015 (1999); S. 
Bar-Shalom and J. Wudka, Phys. Rev. D60, 094016 (1999); J. A. Aguilar-Saavedra, Phys. Lett. B502, 115 (2001). 
A. Heister et al., Phys. Lett. B543, 173 (2002); P. Achard et al., Phys. Lett. B549, 290 (2002); F. D. Aaron et al., Phys. 
Lett. B678, 450 (2009). 

M. Luke and M. J. Savage, Phys. Lett. B307, 387 (1993); T. P. Cheng and M. Sher, Phys. Rev. D35, 3484 (1987). 

A. Fernandez et al., J. Phys. G., in press, arXiv:0911.4995 [hep-ph]. 

B. Aubert et al, Phys. Rev. Lett. 98,211802 (2007). 

M. Staric et al, Phys. Rev. Lett . 98, 211803 (2007). 

Heavy Flavor Averaging Group, [http: / /www. slac. stanford.edu/xorg/hfag/index 
E. Golowich et al, Phys. Rev. D76, 095009 (2007). 
V. Telnov, Nucl. Instr. Meth. A294, 72 (1990). 



